fix: accept Claude native trajectories - #52
Conversation
clawsweeper
Bot
added
rating: 🐚 platinum hermit
|
Codex review: needs maintainer review before merge. Reviewed August 3, 2026, 12:09 AM ET / 04:09 UTC. ClawSweeper reviewWhat this changesThe PR adds Claude Code to ShellBench’s real-native-trajectory allowlist and tests that all pinned native harnesses report real harness events. Merge readinessThis is a focused, source-consistent correction: current main already produces Claude Code trajectories but labels the pinned harness as unsupported. Keep it open for a repository decision because merging changes which completed runs qualify as native benchmark evidence and can affect longitudinal interpretation. Priority: P2 Review scores
Verification
How this fits togetherShellBench runs pinned agent harnesses and records a manifest plus per-task trajectory artifacts for each native evaluation. Aggregation uses the manifest’s trajectory classification and artifacts to decide whether completed results count as native benchmark evidence. flowchart LR
A[Pinned harness] --> B[Native run job]
B --> C[Run manifest]
B --> D[Trajectory artifact]
C --> E[Aggregation checks]
D --> E
E --> F[Eligible native results]
Decision needed
Why: The code change is mechanically narrow, but the allowlist controls benchmark eligibility rather than only display text, so a repository owner should explicitly accept the evidence-policy consequence. Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Confirm that Claude Code stream-json trajectories meet ShellBench’s real-native-evidence contract, then merge this narrow allowlist correction with its regression guard. Do we have a high-confidence way to reproduce the issue? Yes. Current main pins Claude Code as a native harness and has a dedicated trajectory writer, yet the allowlist makes its manifest trajectory mode unsupported; the PR’s focused assertion captures that mismatch. Is this the best way to solve the issue? Yes for the source defect: adding the existing trajectory-producing harness to the real-evidence allowlist is the narrowest repair. The remaining question is whether maintainers accept that eligibility policy. AGENTS.md: not found in the target repository. Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (18 earlier review cycles; latest 8 shown)
|
clawsweeper
Bot
added
proof: sufficient
1 participant
What does this PR do?
Marks Claude Code native trajectories as real harness events so completed Claude runs remain eligible for native aggregation.
Why?
The native runner already converts Claude Code
stream-jsonoutput intotrajectory.json, but the harness allowlist omittedclaude-code. Aggregation therefore classified otherwise valid Claude runs astrajectory_unavailable.Changes
claude-codeto the real-trajectory harness allowlistreal_harness_eventsLive proof
A native Claude Code control run completed four tasks with four result files, four valid completed results, and no infrastructure failures. Aggregation classified every trajectory as
real; the run remained ineligible only because it was intentionally marked as an exploratory four-task subset.4/40.753truerealfor all four tasks0exploratory_subsetTests
python -m pytest -qpasses locally (444 passed, 5 skipped)python -m ruff check clawbench app.py scripts testspasses locally